System and method for selecting targets in an augmented reality environment

ABSTRACT

Techniques are disclosed for facilitating electronic commerce in an augmented reality environment. In some embodiments, a method comprises detecting, by a mobile device, presence of the physical product or the real life service; and presenting, on the mobile device, information to conduct the transaction of a physical product or a real life service via the augmented reality environment. In some embodiments, a method comprises detecting one or more targets in the augmented reality platform using a select area in a perspective of a user, the perspective being captured by a mobile device; and prompting the user to choose an object of interest from the one or more detected targets.

CROSS-REFERENCE TO RELATED APPLICATIONS(S)

This application is a continuation application of prior application Ser.No. 17/101,429, filed on Nov. 23, 2020, which application is acontinuation application of prior application Ser. No. 16/543,011, filedon Aug. 16, 2019, which has issued as U.S. Pat. No. 10,878,636 on Dec.29, 2020, which is a continuation application of prior application Ser.No. 14/939,889, filed on Nov. 12, 2015, which has issued as U.S. Pat.No. 10,388,070 on Aug. 20, 2019, which is continuation of priorapplication Ser. No. 13/874,421, filed on Apr. 30, 2013, which was basedon and claimed priority under 35 U.S.C. § 119(e) of a U.S. Provisionalapplication Ser. No. 61/640,856, filed on May 1, 2012, in the U.S.Patent and Trademark Office, the disclosure of which is incorporated byreference herein in its entirety.

BACKGROUND

Miniaturization of consumer electronics with sophisticated graphicscapabilities and expansive computing power has augmented the activitiesone can engage in via consumer electronics and in particular, portableelectronics such as smart phones, tablets, and the like. Theadvancements in technologies also have brought science fiction intoreality.

Further, portable electronics or other electronics devices now generallyinclude a large variety of sensing capabilities. These capabilities canbe utilized to further enhance the users' augmented reality experiences.

BRIEF DESCRIPTION OF THE DRAWINGS

The present embodiments are illustrated by way of example and are notintended to be limited by the figures of the accompanying drawings. Inthe drawings:

FIG. 1 illustrates an example block diagram of mobile devices able tocommunicate with a host server that generates and controls access tosimulated objects and/or facilitates transactions of a physical productor real life service via an augmented reality environment (e.g., anaugmented reality marketplace) through a network;

FIG. 2 depicts an example block diagram of the components of a hostserver that generates information that facilitates transactions of aphysical product or real life service via an augmented realityenvironment;

FIG. 3A depicts an example functional block diagram of the host serverthat facilitates transactions of a physical product or real life servicevia an augmented reality environment;

FIG. 3B depicts an example block diagram illustrating the components ofthe host server that facilitates transactions of a physical product orreal life service via an augmented reality environment;

FIG. 4A depicts an example functional block diagram of a mobile devicethat presents augmented information with merchandise objects to a userand processes interactions with the objects in the augmented realitymarketplace based on the information;

FIG. 4B depicts an example block diagram of the mobile device thatpresents augmented information with merchandise objects in the augmentedreality marketplace to a user and facilitates user interactions with theobjects based on the information;

FIG. 5A illustrates an example application where mobile devices such asa smart phone or a smart goggle having perspective interfaces which areused to facilitate electronic commerce in an augmented realitymarketplace;

FIG. 5B illustrates an example screenshot on a mobile device displayingan augmented reality environment with objects associated with physicalobjects and/or real life services in a real world environment;

FIG. 6 depicts a flow chart illustrating an example process forfacilitating user interaction with a merchandise object that isidentified in an augmented reality environment;

FIG. 7 depicts a flow chart illustrating an example process forselecting/identifying a merchandise object in an augmented realityenvironment; and

FIG. 8 depicts a diagrammatic representation of a machine in the exampleform of a computer system within which a set of instructions, forcausing the machine to perform any one or more of the methodologiesdiscussed herein, may be executed, according to some embodiments.

The same reference numbers and any acronyms identify elements or actswith the same or similar structure or functionality throughout thedrawings and specification for ease of understanding and convenience.

DETAILED DESCRIPTION

Techniques are disclosed for facilitating electronic commerce in anaugmented reality environment. In some embodiments, a method comprisesdetecting, by a mobile device, presence of the physical product or thereal life service; and presenting, on the mobile device, information toconduct the transaction of a physical product or a real life service viaan augmented reality environment. The mobile device facilitates theaugmented reality environment. In some embodiments, a method comprisesdetecting one or more targets in the augmented reality platform using aselect area in a perspective of a user, the perspective being capturedby a mobile device; and prompting the user to choose an object ofinterest from the one or more detected targets. Among other advantages,embodiments disclosed herein provide an intuitive and integrated userexperience in shopping using augmented reality devices, thereby reducingthe consumer user's effort in conducting such activities, reducingnecessary sales personnel and their working hours, and increasing sales.

Reference in this specification to “one embodiment” or “an embodiment”means that a particular feature, structure, or characteristic describedin connection with the embodiment is included in at least one embodimentof the disclosure. The appearances of the phrase “in one embodiment” invarious places in the specification are not necessarily all referring tothe same embodiment, nor are separate or alternative embodimentsmutually exclusive of other embodiments. Moreover, various features aredescribed which may be exhibited by some embodiments and not by others.Similarly, various requirements are described which may be requirementsfor some embodiments but not other embodiments.

The terms used in this specification generally have their ordinarymeanings in the art, within the context of the disclosure, and in thespecific context where each term is used. Certain terms that are used todescribe the disclosure are discussed below, or elsewhere in thespecification, to provide additional guidance to the practitionerregarding the description of the disclosure. For convenience, certainterms may be highlighted, for example using italics and/or quotationmarks. The use of highlighting has no influence on the scope and meaningof a term; the scope and meaning of a term is the same, in the samecontext, whether or not it is highlighted. It will be appreciated thatsame thing can be said in more than one way.

Consequently, alternative language and synonyms may be used for any oneor more of the terms discussed herein, nor is any special significanceto be placed upon whether or not a term is elaborated or discussedherein. Synonyms for certain terms are provided. A recital of one ormore synonyms does not exclude the use of other synonyms. The use ofexamples anywhere in this specification including examples of any termsdiscussed herein is illustrative only, and is not intended to furtherlimit the scope and meaning of the disclosure or of any exemplifiedterm. Likewise, the disclosure is not limited to various embodimentsgiven in this specification.

Without intent to further limit the scope of the disclosure, examples ofinstruments, apparatus, methods and their related results according tothe embodiments of the present disclosure are given below. Note thattitles or subtitles may be used in the examples for convenience of areader, which in no way should limit the scope of the disclosure. Unlessotherwise defined, all technical and scientific terms used herein havethe same meaning as commonly understood by one of ordinary skill in theart to which this disclosure pertains. In the case of conflict, thepresent document, including definitions will control.

Mobile devices, especially the wearable ones which may be in forms ofeyewear, mobile-enabled wrist watches, head-mounted displays, are nowavailable to provide augmented reality experiences to users. Embodimentsof the present disclosure include systems and methods for facilitatingtransactions of a physical product or real life service via an augmentedreality environment. Additionally, embodiments of the present disclosureinclude systems and methods for enabling selection of a physical productor a real life service in an augmented reality platform via a mobiledevice.

FIG. 1 is a block diagram of an example system 100 including mobiledevices 102A-N able in communication with a host server 124 thatgenerates and controls access to simulated objects and/or facilitatestransactions of a physical product or real life service via an augmentedreality environment through a network 106.

The mobile devices 102A-N can be any system and/or device, and/or anycombination of devices/systems that is able to establish a connectionwith another device, a server and/or other systems. The mobile devices102A-N typically include a display and/or other output functionalitiesto present information and data exchanged between/among the devices102A-N and the host server 124. For example, the mobile devices 102A-Ncan be any of, but are not limited to, a server desktop, a desktopcomputer, a computer cluster, or portable devices including, a notebook,a laptop computer, a handheld computer, a palmtop computer, a mobilephone, a cell phone, a smart phone, a PDA, a Blackberry device, a Treo,an iPhone, cover headsets, heads-up displays, helmet mounted display,head-mounted display, scanned-beam display, wearable computer such asmobile enabled watches, and/or any other mobile interfaces and viewingdevices, etc. Additional examples of mobile devices 102A-N can include aGoogle Glass, smart glasses/goggles, a smart phone, or other suitablemobile computing devices that are configured to facilitate or enable anaugmented reality environment or platform. The mobile devices 102A-N maybe location-aware devices that are able to determine their own locationor identify location information from an external source. In oneembodiment, the mobile devices 102A-N are coupled to a network 106. Insome embodiments, the devices 102A-N and host server 124 may be directlyconnected to one another.

In one embodiment, the host server 124 is operable to provide simulatedobjects (e.g., objects, computer-controlled objects, or simulatedobjects) that correspond to real world physical locations to bepresented to users on mobile devices 102A-N. The simulated objects aretypically software entities or occurrences that are controlled bycomputer programs and can be generated upon request when certaincriteria are met. The host server 124 also processes interactions ofsimulated object with one another and actions on simulated objectscaused by stimulus from a real user and/or the real world environment.

In some embodiments, the host server 124 is operable, in conjunctionwith the mobile devices 102A-N, to facilitate electronic commercialtransactions in an augmented reality environment or platform, such as anaugmented reality marketplace. Services and functions provided by thehost server 124 and the components therein are described in detail withfurther references to the examples of FIG. 3A-3B.

The mobile devices 102A-N are generally operable to provide access(e.g., visible access, audible access) to the simulated objects tousers, for example via user interface 104A-N displayed on the displayunits. The devices 102A-N may be able to detect simulated objects basedon location and/or timing data and provide those objects authorized bythe user for access via the devices. In some embodiments, the mobiledevices 102A-N can facilitate electronic commercial transactions in anaugmented reality environment. In some embodiments, the devices 102A-Ncommunicate with the host server 124 in facilitating the transactions.Services and functions provided by the mobile devices 102A-N and thecomponents therein are described in detail with further references tothe examples of FIG. 4A-4B.

The network 106, over which the mobile devices 102A-N and the hostserver 124 communicate, may be a telephonic network, an open network,such as the Internet, or a private network, such as an intranet and/orthe extranet. For example, the Internet can provide file transfer,remote log in, email, news, RSS, and other services through any known orconvenient protocol, such as, but is not limited to the TCP/IP protocol,Open System Interconnections (OSI), FTP, UPnP, iSCSI, NSF, ISDN, PDH,RS-232, SDH, SONET, etc.

The network 106 can be any collection of distinct networks operatingwholly or partially in conjunction to provide connectivity to the mobiledevices 102A-N and the host server 124 and may appear as one or morenetworks to the serviced systems and devices. In one embodiment,communications to and from the mobile devices 102A-N can be achieved by,an open network, such as the Internet, or a private network, such as anintranet and/or the extranet. In one embodiment, communications can beachieved by a secure communications protocol, such as secure socketslayer (SSL), or transport layer security (TLS).

In addition, communications can be achieved via one or more wirelessnetworks, such as, but is not limited to, one or more of a Local AreaNetwork (LAN), Wireless Local Area Network (WLAN), a Personal areanetwork (PAN), a Campus area network (CAN), a Metropolitan area network(MAN), a Wide area network (WAN), a Wireless wide area network (WWAN),Global System for Mobile Communications (GSM), Personal CommunicationsService (PCS), Digital Advanced Mobile Phone Service (D-Amps),Bluetooth, Wi-Fi, Fixed Wireless Data, 2G, 2.5G, 3G (e.g., WCDMA/UTMSbased 3G networks), 3.5G, 4G, !MT-Advanced, pre-4G, LTE Advanced, mobileWiMax, WiMax 2, WirelessMAN-Advanced networks, enhanced data rates forGSM evolution (EDGE), General packet radio service (GPRS), enhancedGPRS, iBurst, UMTS, HSPDA, HSPA, HSPA+, UMTS-TDD, 1×RTT, EV-DO,messaging protocols such as, TCP/IP, SMS, MIMS, extensible messaging andpresence protocol (XMPP), real time messaging protocol (RTMP), instantmessaging and presence protocol (IMPP), instant messaging, USSD, IRC, orany other wireless data networks or messaging protocols.

The host server 124 may include or be coupled to a user repository 128and/or a simulated object repository 130 and or a merchandise objectrepository 132. The data repositories 128, 130, 132 can store software,descriptive data, images, system information, drivers, and/or any otherdata item utilized by other components of the host server 124 and/or anyother servers for operation. The data repositories 128, 130, 132 may bemanaged by a database management system (DBMS), for example but notlimited to, Oracle, DB2, Microsoft Access, Microsoft SQL Server,PostgreSQL, My SQL, FileMaker, etc.

The repositories 128, 130, 132 can be implemented via object-orientedtechnology and/or via text files, and can be managed by a distributeddatabase management system, an object-oriented database managementsystem (OODBMS) (e.g., ConceptBase, FastDB Main Memory DatabaseManagement System, JDOinstruments, ObjectDB, etc.), an object-relationaldatabase management system (ORDBMS) (e.g., Informix, OpenLink Virtuoso,VMDS, etc.), a file system, and/or any other convenient or knowndatabase management package.

In some embodiments, the host server 124 is able to provide data to bestored in the user data repository 128, the simulated object repository130, the merchandise object repository 132, and/or the can retrieve datastored in the user data repository 128 and/or the simulated objectrepository 130 and/or the merchandise object repository 132.

The user data repository 128 can store user information, userpreferences, user profile, user list of interest, user favorite, userbookmark, user annotation, user tags, access permissions associated withthe users, device information, hardware information, etc. The simulatedobject repository 130 can store software entities (e.g., computerprograms) that control simulated objects and the simulated environmentsin which they are presented for visual/audible access orcontrol/manipulation.

The simulated object repository 130 may further include simulatedobjects and their associated data structures with metadata defining thesimulated object including its associated access permission.

The merchandise object repository 132 can store merchandise informationincluding, for example, merchandise name, alias, attributes, shape,size, dimension, or other physical characteristics or recognitionpatterns or augmented reality markers, location (if suitable),vendor(s), price, SKU code, QR code, I-dimensional or 2-dimensional ormulti-dimensional barcode, RFID code, metadata, tags, annotations, salescontact information.

FIG. 2 depicts an example block diagram of the components of a hostserver 224 that generates information that facilitates transactions of aphysical product or real life service via an augmented realityenvironment.

In the example of FIG. 2 , the host server 224 includes a networkcontroller 202, a firewall 204, a multimedia server 206, a webapplication server 212, an electronic commerce (e-commerce) server 213,and a database including a database storage 216 and database software218.

In the example of FIG. 2 , the network controller 202 can be anetworking device that enables the host server 224 to mediate data in anetwork with an entity that is external to the host server 224, throughany known and/or convenient communications protocol supported by thehost and the external entity. The network controller 202 can include oneor more of a network adaptor card, a wireless network interface card, arouter, an access point, a wireless router, a switch, a multilayerswitch, a protocol converter, a gateway, a bridge, bridge router, a hub,a digital media receiver, and/or a repeater.

The firewall 204, can, in some embodiments, govern and/or managepermission to access/proxy data in a computer network, and track varyinglevels of trust between different machines and/or applications. Thefirewall 204 can be any number of modules having any combination ofhardware and/or software components able to enforce a predetermined setof access rights between a particular set of machines and applications,machines and machines, and/or applications and applications, forexample, to regulate the flow of traffic and resource sharing betweenthese varying entities. The firewall 204 may additionally manage and/orhave access to an access control list which details permissionsincluding for example, the access and operation rights of an object byan individual, a machine, and/or an application, and the circumstancesunder which the permission rights stand.

Other network security functions can be performed or included in thefunctions of the firewall 204, can be, for example, but are not limitedto, intrusion-prevention, intrusion detection, next-generation firewall,personal firewall, etc. without deviating from the novel art of thisdisclosure. In some embodiments, the functionalities of the networkcontroller 202 and the firewall 204 are partially or wholly combined andthe functions of which can be implemented in any combination of softwareand/or hardware, in part or in whole.

In the example of FIG. 2 , the host server 224 includes the multimediaserver 206 or a combination of multimedia servers to manage images,photographs, animation, video, audio content, graphical content,documents, and/or other types of multimedia data for use in or tosupplement simulated content such as simulated objects and theirassociated deployment environment (e.g., a simulated environment). Themultimedia server 206 is any software suitable for delivering messagesto facilitate retrieval/transmission of multimedia data among servers tobe provided to other components and/or systems of the host server 224,for example when rendering a web page, an augmented reality environment,and/or augmenting physical objects using multimedia content (e.g., theaugmented information includes a short video clip introducing physicalproducts or real life services).

In addition, the multimedia server 206 can facilitatetransmission/receipt of streaming data such as streaming images, audio,and/or video. The multimedia server 206 can be configured separately ortogether with the web application server 212, depending on a desiredscalability of the host server 224. Examples of graphics file formatsthat can be managed by the multimedia server 206 include but are notlimited to, ADRG, ADRI, AI, GIF, IMA, GS, JPG, JP2, PNG, PSD, PSP, TIFF,and/or BMP, etc.

The web application server 212 can include any combination of softwareagents and/or hardware modules for accepting Hypertext Transfer Protocol(HTTP) requests from end users, external systems, and/or external mobiledevices and responding to the request by providing the requestors withweb pages, such as HTML documents and objects that can include staticand/or dynamic content (e.g., via one or more supported interfaces, suchas the Common Gateway Interface (CGI), Simple CGI (SCGI), PHP,JavaServer Pages (JSP), Active Server Pages (ASP), ASP.NET, etc.). Inaddition, a secure connection, SSL and/or TLS can be established by theweb application server 212.

The e-commerce server 213 can include any combination of software agentsand/or hardware modules for facilitating electronic commercialtransactions to be initiated, processed, and completed with mobiledevices (e.g., devices 102A-N, FIG. 1 ). The e-commerce server 213typically includes at least one processor and a memory, and may befurther connected to one or more computers (not shown in FIG. 2 forsimplicity) that manage inventory, logistics and/or other commercialfunctions. In some embodiments, the e-commerce server 213 can establishsecured connections using one or more well-known security protocols.

In general, the host server 224 assists, by interactively or proactivelyproviding relevant information, mobile devices such as device 102A-N togenerate the augmented reality user interfaces, which facilitate theaugmented reality marketplace disclosed herein, as shown in the examplescreenshots of FIG. 5B-FIG. 5C. The augmented information, which isprovided by the host server 224 to client users/end devices (e.g., ontothe user interface screens 104A-104N on mobile devices 102A-102N fordisplay). In some embodiments, the web application server 212 and/orthee-commerce server 213 also perform authentication processes beforeresponding to requests for conducting transactions in the augmentedreality marketplace.

The databases 216, 218 can store software, descriptive data, images,system information, drivers, and/or any other data item utilized byother components of the host server for operation. The databases 216,218 may be managed by a database management system (DBMS), for examplebut not limited to, Oracle, DB2, Microsoft Access, Microsoft SQL Server,PostgreSQL, MySQL, FileMaker, etc. The databases 216,218 can beimplemented via object-oriented technology and/or via text files, andcan be managed by a distributed database management system, anobject-oriented database management system (OODBMS) (e.g., ConceptBase,FastDB Main Memory Database Management System, JDOinstruments, ObjectDB,etc.), an object-relational database management system (ORDBMS) (e.g.,Informix, OpenLink Virtuoso, VMDS, etc.), a file system, and/or anyother convenient or known database management package.

In the example of FIG. 2 , the host server 224 includes components(e.g., a network controller, a firewall, a storage server, a multimediaserver, an e-commerce server, and/or a database including a databasestorage and database software, etc.) coupled to one another and eachcomponent is illustrated as being individual and distinct. However, insome embodiments, some or all of the components, and/or the functionsrepresented by each of the components can be combined in any convenientor known manner. Furthermore, the functions represented by the devicescan be implemented individually or in any combination thereof, inhardware, software, or a combination of hardware and software.

FIG. 3A depicts an example functional block diagram of the host server324 that facilitates transactions of a physical product or real lifeservice via an augmented reality environment.

The host server 324 includes a network interface 302, a repositorymanager 304, and/or a search engine 322. The host server 324 is alsocoupled to a user data repository 328 and/or a merchandise objectrepository 332. The user data repository 328 and merchandise objectrepository 332 are described with further reference to the example ofFIG. 1 . The host server 324 can also include a variety of modulesincluding, for example, a pricing management module 312, a physicalcharacteristics management module 314, an annotation management module316, a metadata management module 318, an advertisement/couponmanagement module 320, and/or a recognition/disambiguation processmodule 323. The management modules 312,314, 316,318, and 320 can becoupled to the repository manager 304. The recognition/disambiguationprocess module 323 can be coupled to the search engine 322.

Additional or less modules can be included without deviating from thetechniques discussed in this disclosure. In addition, each module in theexample of FIG. 3A can include any number and combination ofsub-modules, and systems, implemented with any combination of hardwareand/or software modules.

The host server 324, although illustrated as comprised of distributedcomponents (physically distributed and/or functionally distributed),could be implemented as a collective element. In some embodiments, someor all of the modules, and/or the functions represented by each of themodules can be combined in any convenient or known manner. Furthermore,the functions represented by the modules can be implemented individuallyor in any combination thereof, partially or wholly, in hardware,software, or a combination of hardware and software.

In the example of FIG. 3A, the network interface 302 can be a networkingdevice that enables the host server 324 to mediate data in a networkwith an entity that is external to the host server, through any knownand/or convenient communications protocol supported by the host and theexternal entity. The network interface 302 can include one or more of anetwork adaptor card, a wireless network interface card, a router, anaccess point, a wireless router, a switch, a multilayer switch, aprotocol converter, a gateway, a bridge, bridge router, a hub, a digitalmedia receiver, and/or a repeater.

The merchandise object repository 332 can be a database that comprisesany combination of software agents and/or hardware modules able tostore, modify, update, adjust, edit, and/or delete various physical aswell as other attributes of a merchandise object (e.g., a physicalproduct, or a real life service). The management functions of thesephysical and other attributes of merchandise objects can be performed bythe repository manager 304.

A merchandise object can include a physical entity or goods (such as astore, a merchant, commercial product, a living object, an inanimateobject, etc.) or a real life service (such as a restaurant, a spa, abeauty salon, a medical doctor, an art performance, etc.). In someembodiments, merchandise objects can be identified/recognized/detectedusing the various physical and/or other characteristics of the objects,and information relevant to commercial transactions of the physicalproduct or real life service can be presented/displayed/projected in theaugmented reality environment so as to enable the user to conduct suchtransactions.

Characteristics and attributes of merchandise objects that help theidentification/recognition/detection can include what can be perceivedby users in the augmented reality via a physical device (e.g., a mobiledevice or device 102 in the example of FIG. 1 ). For example, they caninclude merchandise name, alias, attributes, shape, size, dimension, orother physical characteristics or recognition patterns or augmentedreality markers, or location (if suitable). They can also include otheridentification information that can be detected by the device 102, suchas vendor(s) name, SKU code, QR code, I-dimensional or 2-dimensional ormulti-dimensional barcode, RFID code.

One embodiment of the host server 324 includes a search engine 322. Thesearch engine 322 can be any combination of software agents and/orhardware modules able to search, detect, and/or identify merchandiseobjects.

More specifically, attributes/clues/characteristics observed by themobile device can be sent to the search engine 322. The search engine322 can perform textual or imagery or other suitable types of search toidentify the merchandise object. The search engine 322 can search ordetect objects either automatically or in response to user request. Forexample, the user can request access to merchandise objects' informationby perform a search request. The search request parameters can include,one or more of, at least a partial name input of the merchandise objectfrom the user, the user's location, the current time or a time period.The search that is performed can automatically detect the merchandiseobject(s) being viewing by or otherwise available for access to the userin the augmented reality. In one embodiment, the merchandise objects arefurther filtered based on preferences or profiles or interest of theuser.

The search engine 322 can further utilize the recognition/disambiguationprocess module 323 to prompt the user to assist in disambiguation of themerchandise object. For example, the module 323 can generate a pluralityof choices in forms of name, picture, locational information, and/ordescription that can help the recognition. For another example, themodule 323 can prompt the user to enter a partial or a full name. Foryet another example, the module 323 can ask the user to take a pluralityof pictures of the merchandise object.

Once identified or detected the presence of the merchandise object orproduct, the host server 324 can access relevant information including,for example, purchase information of the product, price from the vendorof the exact product that the user is viewing in the augmented reality,price from another vendor of the same or similar product, availabilityof the product, any metadata or tags of the product, annotations orreviews of the product added by another user of the augmented realityenvironment, images or video clips that are shared by other users or themerchant, and/or sales contact information. These information can begenerated, maintained and updated using the modules 312-320 coupled tothe repository manager 304 (e.g., by the merchant, by the user, byanother user, or by an administrator of the host server 324).

In some embodiments, the merchandise object is stored in the merchandiseobject repository 332 using a data structure having metadata. Themetadata can properties of the simulated object, location and/or timingparameters (e.g., sale event, availability, store hours, etc.) Thelocation data can be specified with longitude and latitude coordinates,GPS coordinates, and/or relative position. In one embodiment, the objectis associated with a unique identifier or augmented reality (AR)markers. The unique identifier or markers may be further associated witha location data structure having a set of location data that includesthe qualifying location data for the merchandise object. In oneembodiment, the metadata includes a link to data from an external source(e.g., the store's or merchant's website on the Internet).

In one embodiment, this information is automatically provided to theuser when he or she arrives at or near the merchandise object's location(e.g., implicit request) via the device. Alternatively, the informationis provided upon request by the user (e.g., explicit request). Forexample, merchandise objects associated with various attractions, forexample, in a shopping mall or a tourist location in the real world, canbe selected by the user (e.g., via input to the device).

Additionally, an advertisement (e.g., an electronic advertisement) thatis relevant to the merchandise object is delivered by theadvertisement/coupon management module 320 to the user that qualifiesone or more criteria. The user may qualify on a basis of a location,identity, and/or a timing parameter. For example, the user may beprovided with advertisements of local pizza shops or other late nightdining options when the user is driving around town during late nighthours when other dining options may not be available. In anotherexample, an electronic coupon can be presented to a user using thedevice based on the merchandise object identification result, forexample, when the device is located at the location during a certainperiod of time that satisfies the criteria. The electronic coupon may beredeemed by the user for conducting transactions of the physical productor the real life service.

FIG. 3B depicts an example block diagram illustrating the components ofthe host server 324 that facilitates transactions of a physical productor real life service via an augmented reality environment.

In one embodiment, host server 334 includes a network interface 302, aprocessing unit 334, a memory unit 336, a storage unit 338, a locationsensor 340, and/or a timing module 342. Additional or less units ormodules may be included. The host server 324 can be any combination ofhardware components and/or software agents for generating informationfor merchandise objects in facilitating transactions in an augmentedreality marketplace environment. The network interface 302 has beendescribed in the example of FIG. 3A.

One embodiment of the host server 324 further includes a processing unit334. The data received from the network interface 302, location sensor340, and/or the timing module 342 can be input to a processing unit 334.The location sensor 340 can include GPS receivers, RF transceiver, anoptical rangefinder, etc. The timing module 342 can include an internalclock, a connection to a time server (via NTP), an atomic clock, a GPSmaster clock, etc.

The processing unit 334 can include one or more processors, CPUs,microcontrollers, FPGAs, ASICs, DSPs, or any combination of the above.Data that is input to the host server 334 can be processed by theprocessing unit 334 and output to a display and/or output via a wired orwireless connection to an external device, such as a mobile phone, aportable device, a host or server computer by way of a communicationscomponent.

One embodiment of the host server 324 further includes a memory unit 336and a storage unit 338. The memory unit 335 and a storage unit 338 are,in some embodiments, coupled to the processing unit 334. The memory unitcan include volatile and/or non-volatile memory. In receivingcharacteristics of and generating relevant information for themerchandise objects, the processing unit 334 may perform one or moreprocesses related to identifying/recognizing/detecting the merchandiseobjects and/or generating information for conducting transactions ofsuch objects.

In some embodiments, any portion of or all of the functions described ofthe various example modules in the host server 324 of the example ofFIG. 3A can be performed by the processing unit 334.

FIG. 4A depicts an example functional block diagram of a mobile device402 that presents augmented information with merchandise objects to auser and processes interactions with the objects in the augmentedreality marketplace.

The mobile device 402 includes a network interface 404, a timing module406, an RF sensor 407, a location sensor 408, an image sensor 409, anobject identifier module 412, a user selection module 414, a userstimulus sensor 416, a motion/gesture sensor 418, an environmentalstimulus sensor 420, an audio/video output module 422, and/or othersensors 410. The mobile device 402 may be any electronic device such asthe devices described in conjunction with the mobile devices 102A-N inthe example of FIG. 1 including but not limited to portable devices, acomputer, a server, location-aware devices, mobile phones, PDAs,laptops, palmtops, iPhones, cover headsets, heads-up displays, helmetmounted display, head-mounted display, scanned-beam display, GoogleGlass, smart glasses/goggles, wearable computer such as mobile enabledwatches or eyewear, and/or any other mobile interfaces and viewingdevices, etc.

In one embodiment, the mobile device 402 is coupled to a merchandiseobject repository 431 for identifying/detecting/recognizing merchandiseobjects (e.g., physical product or real life service) in the augmentedreality environment. The merchandise object repository 431 may beinternal to or coupled to the mobile device 402 but the contents storedtherein can be illustrated with reference to the example of amerchandise object repository 132 described in the example of FIG. 1 .

Additional or less modules can be included without deviating from thenovel art of this disclosure. In addition, each module in the example ofFIG. 4A can include any number and combination of sub-modules, andsystems, implemented with any combination of hardware and/or softwaremodules.

The mobile device 402, although illustrated as comprised of distributedcomponents (physically distributed and/or functionally distributed),could be implemented as a collective element. In some embodiments, someor all of the modules, and/or the functions represented by each of themodules can be combined in any convenient or known manner. Furthermore,the functions represented by the modules can be implemented individuallyor in any combination thereof, partially or wholly, in hardware,software, or a combination of hardware and software.

In the example of FIG. 4A, the network interface 404 can be a networkingdevice that enables the mobile device 402 to mediate data in a networkwith an entity that is external to the host server, through any knownand/or convenient communications protocol supported by the host and theexternal entity. The network interface 404 can include one or more of anetwork adaptor card, a wireless network interface card, a router, anaccess point, a wireless router, a switch, a multilayer switch, aprotocol converter, a gateway, a bridge, bridge router, a hub, a digitalmedia receiver, and/or a repeater.

According to the embodiments disclosed herein, the mobile device 402 canfacilitate transactions (e.g., e-commerce transactions) of a merchandiseobject (e.g., a physical product or a real life service) via anaugmented reality environment provided by the mobile device 402. Themobile device 402 can detect presence of the physical product or thereal life service when the product or service is seen/viewed/looked atby the user or in vicinity of the user. In some embodiments, the mobiledevice 402 automatically identifies the product or service from aperspective of a user. The perspective can be captured or sensed by themobile device 402 via, for example, a camera or an image sensor.

After detecting/identifying/recognizing the merchandise object, themobile device 402 can access information relevant to the transaction ofthe merchandise object(s) from a merchandise object repository (e.g.,repository 431 in mobile device 402 and/or repository 332 in host server324), and can present the relevant information to the user to conductthe transaction of the product or service via the augmented realityenvironment. For example, the mobile device 402 can render on a displayof the device 402

More specifically, one embodiment of the mobile device 402 includes atiming module 406. The timing module 406 can be any combination ofsoftware agents and/or hardware modules able to identify, detect,transmit, compute, a current time, a time range, and/or a relative timeof a request related to simulated objects/environments.

The timing module 406 can include a local clock, timer, or a connectionto a remote time server to determine time. The timing module 406 can beimplemented via any known and/or convenient manner including but notlimited to, electronic oscillator, clock oscillator, or various types ofcrystal oscillators. According to some embodiments, the timing module406 can provide time data which may be used to assist identification,recognition or detection by the mobile device 402 of the merchandiseobject. Examples that the mobile device 402 can consider include thetime of day of a request, the date of the request, a relative time toanother event, the time of year of the request, and/or the time span ofa request or activity pertaining to merchandise objects.

One embodiment of the mobile device 402 includes a radio frequency (RF)or other types of wireless sensor 407. The RF sensor 407 can be anycombination of software agents and/or hardware modules able to detect oridentify an RF identification chip or other similar wirelessidentification tags.

One embodiment of the mobile device 402 includes a location sensor 408.The location sensor 408 can be any combination of software agents and/orhardware modules able to identify, detect, transmit, compute, a currentlocation, a previous location, a range of locations, a location at or ina certain time period, and/or a relative location of the mobile device402. The location sensor 408 can include a local sensor (e.g., GPSsensor) or a connection to an external entity (e.g., a base station) todetermine the location information. The location sensor 408 candetermine location or relative location of the mobile device 402 via anyknown or convenient manner including but not limited to, GPS, cell phonetower triangulation, mesh network triangulation, relative distance fromanother location or device, RF signals, RF fields, optical range findersor grids, etc. The location sensor 408 can identify location data anddetermine whether the location data satisfies the location parameter ofa certain criteria. In some embodiments, the location sensor 408provides location data to the host server (e.g., host server 324 of FIG.3A) for the host server to determine whether the criteria is satisfied.

One embodiment of the mobile device 402 includes a camera or an imagesensor 409. The image sensor 409 can be any combination of softwareagents and/or hardware modules able to record or capture still imagessuch as photographs or moving images such as video clips or movies.

It is noted that a person having ordinary skills in the art willunderstand that sensor modules 407-409 are exemplary, as the mobiledevice 402 can include other sensors 410 including, for example,accelerometers, gyroscopic gauges, proximity sensors, light sensors,microphones, temperature sensors, heart beat sensors, touch sensitivesensors including touch-screens, fingerprint sensors, and so forth.

The mobile device 402 utilizes suitable sensors to detect presence ofthe physical product or the real life service in the augmented realityenvironment or platform. In some embodiments, the mobile deviceautomatically identifies or detects the product(s) or service(s) thatfalls within a perspective of the user, that is, being viewed/lookedat/seen by the user. The product is recognized/identified/detected byusing one of several mechanisms as employed by the object identifiermodule 412, and by querying the merchandise object repository 431.

More specifically, mobile device 402 includes the object identifiermodule 412. The object identifier module 412 can be any combination ofsoftware agents and/or hardware modules able to identify, detect, and/orrecognize, merchandise objects for presentation to a user. The objectidentifier module 412, in one embodiment, is coupled to the timingmodule 406, the RF sensor 407, the location sensor 408, the image sensor409, and/or other sensors 410. The identifier module 412 can alsoreceive data signals from the motion/gesture sensor 418, the userstimulus sensor 416, the environmental stimulus sensor 420, and iscoupled to the merchandise object repository 431 via the networkinterface 404. The object identifier module 412 is operable to identifythe merchandise objects available for access using the device 402.

In some embodiments, the presence of the product is detected oridentified in the augmented reality environment by a shape or otherphysical characteristics for presentation via the mobile device tofacilitate transactions. For example, the product or service can have anunique combination of shape, size, color, or other physical dimension orcharacteristics (as registered in the merchandise object repository 431)so that the object identifier module 412 can recognize it. Such shape orphysical characteristics can be detected from, for example, the imagesensor 409. As previously mentioned, the entries in the repository 431can also include recognition patterns or photos or videos or metadatathat may help identifying the product.

In some embodiments, the presence of the product or service is detectedor identified in the augmented reality environment by a geographiclocation of the product or service. For example, the location of themobile device 402 can assist in identifying (e.g., by pinpointing orruling out) certain product or service that is only available in thatlocation. Such location data can be detected from, for example, thelocation sensor 408.

In some embodiments, the presence of the product is detected oridentified in the augmented reality environment by one or more of: (i) avisual marker; (ii) a marker or tag; (iii) a one-dimensional barcode; or(iv) a multi-dimensional barcode, on the product. For example, a markeron the object such as a QR code or other augmented reality marker can bepresented for identification or detection (e.g., via the image sensor409). In another example, a barcode representing a stock-keeping unitnumber (SKU) may be present. The barcode can be one-dimensional ormulti-dimensional.

In some embodiments, the presence of the product is detected oridentified in the augmented reality environment by a radio frequencyidentification (RFID) chip coupled to the product. For example, the RFIDchip can be embedded in the product, or on a label attached to theproduct, or attached on a surface or a nearby location of the product.The RFID chip can be detected and read, for example, via the RFsensor/RFID reader 407.

In some embodiments, the presence of the product is detected oridentified in the augmented reality environment by optical characterrecognition (OCR) of written materials on or nearby the product. Forexample, the image sensor 409 can capture text on labels or othermarketing materials on or near the product, and the object identifiermodule 412 can use well-known OCR techniques to convert the writtenmaterials into text for identification (e.g., via a search using thesearch engine 322 to search merchandise repositories 332, 431).

Further, as an additional or an alternative embodiment, the objectidentifier module 412 can receive input from the user to disambiguate anidentity of the product or service. For example, the user's input caninclude at least partially a name of the product or service. The usercan be prompted to name the product in assisting the disambiguationprocess proactively or interactively. Such user input can be receivedfrom, for example, the motion/gesture sensor 418 or the user stimulussensor 416.

Some embodiments of the object identifier module 412 can also receiveadvisory data associated with the product or service to assist theidentifying/recognizing/detecting the product or service. The advisorydata being added by another user of the augmented reality environment,or by an administrator or merchant of the augmented reality marketplace.The advisory data can include one or more of: (i) metadata or tagsrelevant to the product or service, or (ii) an annotation that isdescriptive of the product or service. The metadata about the productcan be added by another previous user who already identified theproduct, and/or tagged the product at the location that the mobiledevice 402 is currently situated.

After identifying/detecting/recognizing merchandise objects, externalactions can be performed by the user in the real world to interact withthe detected merchandise objects. In one embodiment, such actions orrequests from the user to perform a requested action on a merchandiseobject can be received by the user stimulus sensor 416. In oneembodiment, the motion/gesture sensor 418 is operable to detect motionof the device 402. The detected motion is used by the user selectionmodule 414 to perform actions responsive to the presented informationaccording to the detected motion of the device. Motion detecting caninclude detecting velocity and/or acceleration of the device 402 or agesture of the user handling the device 402. The motion/gesture sensor418 can include for example, an accelerometer.

More specifically, with the merchandise object(s) identified, the mobiledevice 402 can present (e.g., using the user selection module 414 and/oraudio/video output module 422) information relevant to the merchandiseproduct or service to the user so that the user can initiate variousactions responsive to the presented information, for example, initiatingthe transaction of the product or service via the mobile device 402.Other example actions can include, for example, get price, compareprice, read reviews, get availability, add to shopping cart, call asales representative (e.g., a local sales person or a virtual one) forassistance, buy the product, tag the product or add metadata for theproduct, share the product with another user, or bookmark the productfor later consideration. The relevant information, including transactioninformation, can be transmitted from the repository 431, which may belocal or may be hosted by an online server (e.g., host server 324,224,which may also include or be coupled to e-commerce server 213) thatenables electronic commerce.

In some embodiments, a user interface generator in the user selectionmodule 414 can be used to create buttons/links/etc. that are selectableby the user. The interactions or actions from the user can be detectedor determined using the user selection module 414 (e.g., viainterpreting signals generated from the motion/gesture sensor 418, userstimulus sensor 416, and/or the environmental stimulus sensor 420).

Any response to the user's selection or action can be reflected (e.g.,highlighting a selected merchandise, blinking of a selected button,displaying an annotation or review by another user, or displaying ashort video clip of the selected merchandise) in the augmented realityenvironment via the audio/video output module 422.

In some embodiments, the information is rendered translucently anddisposed adjacently or partially overlaid with the product or servicedepicted in the augmented reality environment on the mobile device 402.In some embodiments, the information includes advisory data associatedwith the product or service. Examples of advisory data can includemetadata or tags that are deemed (either automatically by the objectidentifier module 412, or by another user, or by a merchant or anadministrator) relevant to the product or service, or (ii) an annotationthat is descriptive of the product or service.

The mobile device 402 can also prompt the user to enter an advisory datafor the product or service by, for example, operating a virtual keyboardusing the user stimulus sensor 416 and/or motion/gesture sensor 418.Then, the user can select to share (e.g., via network interface 404) theentry to another user of the augmented reality environment.

The content of the interactions can also be decided based onenvironmental factors including, for example, physical location of theuser and/or the merchandise, user action, userfeedback/annotation/review/tags/etc. as provided by the user of thedevice 402 or another user of the augmented reality platform usinganother device and/or at another physical location, motion/movement ofthe device 402, gesture of the user using the device 402.

In some embodiments, an advertisement relevant to the product or servicecan be identified and displayed in the augmented reality environment viathe mobile device 402. The advertisement is identified or detected basedon one or more of: (i) location, (ii) time, (iii) interest or profile ofa user of the mobile device 402. For example, the environmental stimulussensor 420 can detect environmental factors or changes in environmentalfactors surrounding the real environment in which the device 402 islocated, and the advertisement can be provided based on these factors orchanges. Environmental factors can include, weather, temperature,topographical characters, density, surrounding businesses, buildings,living objects, etc. These factors or changes in them can also affectthe positioning and/or characters of the presented information for themerchandise objects in the augmented reality in which they are presentedto a user via the device 402.

In some embodiments, the advertisement is identified or detected basedon a pointing direction of the mobile device 402. In some embodiments,the advertisement is identified or detected based on velocity or speedof movement of the mobile device, or based on an ambient temperature inwhich the mobile device physically resides, or based on a plurality ofsensors (e.g., sensors 407-410) on the mobile device 402. Further, theadvertisement is identified or detected (e.g., by a combined reading andinterpretation of data signals from the location sensor 408, the imagesensor 409, and a directional sensor which may be included among othersensors 410) based on a landmark which the mobile device is pointed ator in a vicinity of.

In some embodiments, the information includes a contact information of asales representative of the product or service, and the mobile device402 prompts the user to contact the sale representative based on thecontact information.

Additionally, in some embodiments, the mobile device 402 allows the userto bookmark the product or service, and in some embodiments, the mobiledevice 402 allows the user to share the bookmark with another user ofthe augmented reality environment.

According to some additional or alternative embodiments, the mobiledevice 402, in providing the augmented reality environment, can enablethe user to select a physical product or service that is in theaugmented reality. The mobile device 402 can detect one or more targetsin the augmented reality platform using a select area on a perspectiveof a user. The perspective can be captured, for example, by a camera ofthe mobile device 402. After the targets are detected, the mobile device402 prompts the user to choose an object of interest from the one ormore detected targets.

More specifically, the mobile device 402 can detect one or more targetsin the augmented reality platform using a select area in a perspectiveof a user, and prompt the user to choose an object of interest from theone or more detected targets. As previously mentioned, the perspectivecan be captured by the mobile device 402 (e.g., via the image sensor409. For example, just as a user using a mouse to select items on apersonal computer's screen, some embodiments disclosed herein enableselection of real and/or virtual items in an augmented realityenvironment/platform/marketplace, which in turn enables actions to betaken in relation to the selected items (e.g., the aforementionedtransactions of or interactions with the merchandise objects).

The detection of targets can be performed, for example, by objectidentifier module 412 of the mobile device 402. According to someembodiments, the detection of targets in the augmented realityenvironment can be performed by pointing the device 402 at thetarget(s). In some embodiments, the detection of targets can beperformed by moving a pointer or a select area on the display of thedevice 402 to point at or frame the object in a reticle or a circular orrectangular frame (e.g., select area 520, 525, described below withrespect to FIG. 5A). It is noted that the select area can be anysuitable shape and sizes. In some embodiments, the mobile device 402 canprovide (e.g., via audio/video output module 422 to a display of themobile device 402) the user with a lasso or a selection tool in theperspective to surround a respective target so as to form the selectarea. Additionally, the mobile device 402 can prompt the user to choosethe object of interest from a set of choices such as a number of targetsthat are recognized in the perspective.

In some instances, for example, when target is not recognizedautomatically or when the user so desires, the mobile device 402 canprompt the user, or the user can select, to enter a learning mode inwhich the mobile device 402 can receive assistance from the user toperform the detection or recognition of the target. The assistance caninclude one or more of (i) entering at least partially a name, (ii)entering metadata, (iii) defining a boundary via a pointer, or (iv)taking snapshots from a plurality of views to develop a recognitionpattern, for a respective target. For example, the teaching can be doneby aiming at the target or circling around the target with the interface(e.g., using the select area or the lasso tool, provided by theinterface generator module in the user selection module 414) on thedevice 402, and then by selecting a “learn” function or button on theinterface in the augmented reality environment. Then, the mobile device402 can ask the user for a name or other metadata for the target, andmay ask for assistance in defining the boundaries of the object (e.g.,via the lasso tool). The device 402 can also ask the user to take aseries of snapshot views (e.g., via camera or image sensor 409) of thetarget to develop a pattern recognition database for it, as well aspotentially a series of tests to improve recognition.

After targets are recognized, they can then be selected by the user foractions or interactions. The mobile device 402 can receive the user'schoice of the object of interest by detecting movement and selection bya selection tool or a pointer, or movement and selection represented bya gesture (e.g., via the motion/gesture sensor 418, user stimulus sensor416, and/or hand/finger/other gestures captured by the image sensor409).

More specifically, in detecting the user's gesture to move a pointer ortargeting or selection tool, and/or to select an object, the mobiledevice 402 can perform capturing, via its various sensors, the gesturefrom one or more of: (i) movements or non-movements of an eye of theuser, (ii) locations of a focal point of an eye of the user, or (iii)movements of an eye lid of the user. Additionally, the mobile device 402can capture the gesture from one or more of: (i) movements of hand orfinger as recognized by a camera of the mobile device, (ii) movements ofa virtual pointer controlled by a wireless peripheral device, (iv)movements of a virtual pointer controlled by a touch-sensitive displayof the mobile device, (v) movements of the mobile device itself, or (vi)movements of the user's head, limbs, or torso. The capturing can befurther based on a speed or velocity of the movements. As such, in lightof the above, the present embodiments can capture or identify gesturesfrom, for example, winking of the user and/or an eye focus or eye fociof the user. Another example of gesture controlling can include fingeror arm gesturing as captured by camera and/or distancedetector/proximity detectors, so that the user can perform “spatial” or“virtual” gesturing in the air or other detectable spaces with similargestures as those well-known gestures applicable to a mobile phone'stouch screen. Yet another example of gesture controlling can include eyeball motion tracking and/or eye focal point tracking. In this way, theuser of mobile device 402 may operate various selection mechanisms, forexample, using his or her eyes (e.g., via eye movement tracking) or bymoving his or her hands/arms/fingers in the perspective to make specificgestures such as pointing or tracing the outline of some object, or byoperating a virtual pointer in the scene using a handheld peripheralsuch as a wireless pointing device or a mouse equivalent, or by touchinga touch-sensitive display on a mobile device and gesturing on it toindicate actions and selections, or by moving the device itself withspecific gestures and velocities to use the device as a pointer orselection tool. Additional gestures may include eye tracking anddetermining a focus of the eye for targeting things, and/or blinking toselect a target that is in the focal point, to take a photo, or toselect a button, etc.

After receiving the user's selection or choice of the object ofinterest, the mobile device 402 can optionally confirm with the user ofthe choice of the object of interest. The confirmation can includehighlighting or outlining the target in the augmented reality platformby employing, for example, the audio/video output module 422.

FIG. 4B depicts an example block diagram of the mobile device 402 thatpresents augmented information with merchandise objects in the augmentedreality marketplace to a user and facilitates user interactions with theobjects.

In one embodiment, mobile device 402 includes a network interface 432, aprocessing unit 434, a memory unit 436, a storage unit 438, a locationsensor 440, an accelerometer/motion sensor 442, an audio outputunit/speakers 446, a display unit 450, an image capture unit 452, apointing device/sensor 454, a input device 456, and/or a touch screensensor 458. Additional or less units or modules may be included. Themobile device 402 can be any combination of hardware components and/orsoftware agents for that presenting information with respect tomerchandise objects to a user and facilitating user interactions withthe merchandise objects. The network interface 432 has been described inthe example of FIG. 4A.

One embodiment of the mobile device 402 further includes a processingunit 434. The location sensor 440, accelerometer/motion sensor 442, andtimer 444 have been described with reference to the example of FIG. 4A.

The processing unit 434 can include one or more processors, CPUs,microcontrollers, FPGAs, ASICs, DSPs, or any combination of the above.Data that is input to the mobile device 402 for example, via the imagecapture unit 452, pointing device/sensor 554, input device 456 (e.g.,keyboard), and/or the touch screen sensor 458 can be processed by theprocessing unit 434 and output to the display unit 450, audio outputunit/speakers 446 and/or output via a wired or wireless connection to anexternal device, such as a host or server computer that generates andcontrols access to simulated objects by way of a communicationscomponent.

One embodiment of the mobile device 402 further includes a memory unit436 and a storage unit 438. The memory unit 436 and a storage unit 438are, in some embodiments, coupled to the processing unit 434. The memoryunit can include volatile and/or non-volatile memory. In generating andcontrolling access to the simulated objects, the processing unit 434 mayperform one or more processes related to presenting information withrespect to merchandise objects to a user and/or facilitating userinteractions with the merchandise objects based on the presentedinformation.

In some embodiments, any portion of or all of the functions described ofthe various example modules in the mobile device 402 of the example ofFIG. 4A can be performed by the processing unit 434. In particular, withreference to the mobile device illustrated in FIG. 4A, various sensorsand/or modules can be performed via any of the combinations of modulesin the control subsystem that are not illustrated, including, but notlimited to, the processing unit 434 and/or the memory unit 436.

FIG. 5A illustrates an example application where mobile devices 502 suchas a smart phone 502A or a smart goggle 502B having perspectiveinterfaces 510,515 which can be used to facilitate electronic commercein an augmented reality marketplace.

With the embodiments disclosed herein, for example, when the user isdoing “window shopping,” that is, when the user is looking at amerchandise product in a store window, or on a shelf, or when the useris near a physical store, the user can utilize the mobile devices 502A,502B to access information 504A, 504B regarding theidentified/detected/recognized merchandise objects. The merchandiseobjects can be selected using select areas 520, 525 in ways that arediscussed above. Then, the user can select to perform actions 506(including transactions) to the merchandise objects in ways that arediscussed above. Thus, for example, through the perspective interface510, 515 in the augmented reality environment, the user can get priceof, add comments/annotations/tags for, read reviews of, get availabilityof, or select to purchase the selected merchandise object. The objectscan be identified by ways discussed above including, for example, abarcode 508 attached to the merchandise object. Written materials foridentifying objects and/or other relevant information (such as salesevent information 530 as illustrated in FIG. 5A) can also berecognized/detected by the mobile devices 502A, 502B.

Other example augmented information can be provided to the user toinform the user if the store is now open, what is on sale now, etc.Similarly, other sales information including advertisement, video clips,images, or sales representative's contact information can be presentedor otherwise made accessible to the user. The information can alsoinclude entries from other users of the augmented reality, including butnot limited to, their names and profiles, recent blog or microblogposts, recent photos or videos or links or documents added or annotatedby them, their recent locations, their professions, etc.

FIG. 5B illustrates an example screenshot on a device 502 displaying anaugmented reality environment 520 with objects 522 (includingmerchandise objects) associated with physical objects and/or real lifeservices 526 in a real world environment 524.

As aforementioned, augmented reality environment 520 reflects aperspective of the user as captured by the mobile device 502 (e.g., viaa camera of the device 502). As such, the display and movement of theobject 522 in the augmented reality environment 520 corresponds to thedisplay and movement of the physical object 526 (e.g., car) in the realworld 524.

In augmented reality environment 502, relevant information regarding thephysical object 526 can be rendered or presented with the object 522 soas to “augment” the object 522. Such information or data can be aboutthe people or entities that are in or near a particular geographicallocation. Further, the mobile device 502 can facilitate transactions ofthe real physical product or service 526 based on the information, sothat transactions can be conducted by users via devices when the user isphysically near or at the physical product or the real life service.

FIG. 6 depicts a flow chart illustrating an example process 600 forfacilitating user interaction with a merchandise object that isidentified in an augmented reality environment. With reference to FIGS.1, 3A, 4A, and SA, process 600 is now described.

First, a mobile device (e.g., devices 102A-102N, FIG. 1 ; device 402,FIGS. 4A and 4B; devices 502A, 502B, FIG. 5A) detects (610) presence ofa physical product or a real life service. The mobile device 102, 402,502 provides the augmented reality environment (e.g., throughperspective interfaces 510,515, FIG. 5A) to the user. In someembodiments, the device 102,402, 502 automatically identifies (615) theproduct or service from the perspective 510, 515 of the user captured bythe mobile device (e.g., via image sensor 409, FIG. 4A).

Then, the mobile device 102,402, 502 can query a merchandise objectrepository (e.g., repository 132, FIG. 1 ; repository 332, FIG. 3A;repository 431, FIG. 4A) in identifying the merchandise product orservice and obtaining relevant information. Thereafter, the mobiledevice 102,402, 502 presents (620), on the mobile device (e.g., on userinterface 104A-104N, FIG. 1 , or on perspective 510,515), information(e.g., information 504A, 504B, FIG. 5A) to conduct the transaction ofthe product or service via the augmented reality environment. In someembodiments, the device 102, 402, 502 renders (625), on a display of themobile device (e.g., via audio/video output module 422, FIG. 4A), atransaction information associated with the product or service.

Next, in some embodiments, the mobile device 102,402, 502 can receive(630) input from the user to disambiguate an identity of the product orservice. Further, in some embodiments, the mobile device 102,402, 502can receive (640) advisory data associated with the product or service.The advisory data can be added, for example, by another user of theaugmented reality environment.

After the information is presented, the mobile device 102,402, 502 caninitiate (650) the transaction of the product or service via the mobiledevice based on responses to the presented information (e.g., asdetected via various sensors 406-410, 416,418,420 of device 402 anddetermined by the user selection module 414, FIG. 4A) from the user.

FIG. 7 depicts a flow chart illustrating an example process 700 forselecting/identifying a merchandise object in an augmented realityenvironment. With reference to FIGS. 1, 3A, 4A, and SA, process 700 isnow described.

First, a mobile device (e.g., devices 102A-102N, FIG. 1 ; device 402,FIGS. 4A and 4B; devices 502A, 502B, FIG. 5A) detects (710) one or moretargets in an augmented reality platform using a select area (e.g.,select areas 520, 525, FIG. 5A) in a perspective (e.g., perspective 510,515, FIG. 5A) of a user. The perspective 510, 515 is captured by themobile device 102, 402, 502 (e.g., via image sensor 409, FIG. 4A). Insome embodiments, the mobile device 102,402, 502 provides (712), via adisplay of the mobile device (e.g., via audio/video output module 422,FIG. 4A), the user with a lasso tool in the perspective 510, 515 tosurround a respective target so as to form the select area 520, 525. Inaccordance with some embodiments, the mobile device 102,402, 502 prompts(714), via audio/video output module 422, the user to enter a learningmode in which the mobile device 102,402, 502 receives (714) anassistance from the user to perform the detection.

After detecting the targets, the mobile device 102,402, 502 prompts(720) the user to choose an object of interest from the one or moredetected targets. Then, the mobile device 102,402,502 receives (730) theuser's choice of the object of interest (e.g., via motion/gesture sensor418, user stimulus sensor 416, and/or other gestures captured by imagesensor 409, as determined by the user selection module 414, FIG. 4A).The mobile device 102,402, 502 can detect (732) movement and selectionby a selection tool or a pointer, and/or movement and selectionrepresented by a gesture. Additionally, the mobile device 102,402, 502can confirm (740) with the user of the choice of the object of interest.

FIG. 8 shows a diagrammatic representation of a machine in the exampleform of a computer system 800 within which a set of instructions, forcausing the machine to perform any one or more of the methodologiesdiscussed herein, may be executed.

In alternative embodiments, the machine operates as a standalone deviceor may be connected (e.g., networked) to other machines. In a networkeddeployment, the machine may operate in the capacity of a server or aclient machine in a client-server network environment, or as a peermachine in a peer-to-peer (or distributed) network environment.

The machine may be a server computer, a client computer, a personalcomputer (PC), a tablet PC, a set-top box (STB), a personal digitalassistant (PDA), a cellular telephone, a web appliance, a networkrouter, switch or bridge, or any machine capable of executing a set ofinstructions (sequential or otherwise) that specify actions to be takenby that machine.

While the machine-readable medium is shown in an exemplary embodiment tobe a single medium, the term “machine-readable medium” should be takento include a single medium or multiple media (e.g., a centralized ordistributed database, and/or associated caches and servers) that storethe one or more sets of instructions. The term “machine-readable medium”shall also be taken to include any medium that is capable of storing,encoding or carrying a set of instructions for execution by the machineand that cause the machine to perform any one or more of themethodologies of the present disclosure.

In general, the routines executed to implement the embodiments of thedisclosure, may be implemented as part of an operating system or aspecific application, component, program, object, module or sequence ofinstructions referred to as “computer programs.” The computer programstypically comprise one or more instructions set at various times invarious memory and storage devices in a computer, and that, when readand executed by one or more processors in a computer, cause the computerto perform operations to execute elements involving the various aspectsof the disclosure.

Moreover, while embodiments have been described in the context of fullyfunctioning computers and computer systems, those skilled in the artwill appreciate that the various embodiments are capable of beingdistributed as a program product in a variety of forms, and that thedisclosure applies equally regardless of the particular type of machineor computer-readable media used to actually effect the distribution.

Further examples of machine or computer-readable media include but arenot limited to recordable type media such as volatile and non-volatilememory devices, floppy and other removable disks, hard disk drives,optical disks (e.g., Compact Disk Read-Only Memory (CD ROMS), DigitalVersatile Disks, (DVDs), etc.), among others, and transmission typemedia such as digital and analog communication links.

Unless the context clearly requires otherwise, throughout thedescription and the claims, the words “comprise,” “comprising,” and thelike are to be construed in an inclusive sense, as opposed to anexclusive or exhaustive sense; that is to say, in the sense of“including, but not limited to.” As used herein, the terms “connected,”“coupled,” or any variant thereof, means any connection or coupling,either direct or indirect, between two or more elements; the coupling ofconnection between the elements can be physical, logical, or acombination thereof. Additionally, the words “herein,” “above,” “below,”and words of similar import, when used in this application, shall referto this application as a whole and not to any particular portions ofthis application. Where the context permits, words in the above DetailedDescription using the singular or plural number may also include theplural or singular number respectively. The word “or,” in reference to alist of two or more items, covers all of the following interpretationsof the word: any of the items in the list, all of the items in the list,and any combination of the items in the list.

The above detailed description of embodiments of the disclosure is notintended to be exhaustive or to limit the teachings to the precise formdisclosed above. While specific embodiments of, and examples for, thedisclosure are described above for illustrative purposes, variousequivalent modifications are possible within the scope of thedisclosure, as those skilled in the relevant art will recognize. Forexample, while processes or blocks are presented in a given order,alternative embodiments may perform routines having steps, or employsystems having blocks, in a different order, and some processes orblocks may be deleted, moved, added, subdivided, combined, and/ormodified to provide alternative or subcombinations. Each of theseprocesses or blocks may be implemented in a variety of different ways.Also, while processes or blocks are at times shown as being performed inseries, these processes or blocks may instead be performed in parallel,or may be performed at different times. Further any specific numbersnoted herein are only examples: alternative implementations may employdiffering values or ranges.

The teachings of the disclosure provided herein can be applied to othersystems, not necessarily the system described above. The elements andacts of the various embodiments described above can be combined toprovide further embodiments.

Any patents and applications and other references noted above, includingany that may be listed in accompanying filing papers, are incorporatedherein by reference. Aspects of the disclosure can be modified, ifnecessary, to employ the systems, functions, and concepts of the variousreferences described above to provide yet further embodiments of thedisclosure.

These and other changes can be made to the disclosure in light of theabove Detailed Description. While the above description describescertain embodiments of the disclosure, and describes the best modecontemplated, no matter how detailed the above appears in text, theteachings can be practiced in many ways. Details of the system may varyconsiderably in its implementation details, while still beingencompassed by the subject matter disclosed herein. As noted above,particular terminology used when describing certain features or aspectsof the disclosure should not be taken to imply that the terminology isbeing redefined herein to be restricted to any specific characteristics,features, or aspects of the disclosure with which that terminology isassociated. In general, the terms used in the following claims shouldnot be construed to limit the disclosure to the specific embodimentsdisclosed in the specification, unless the above Detailed Descriptionsection explicitly defines such terms. Accordingly, the actual scope ofthe disclosure encompasses not only the disclosed embodiments, but alsoall equivalent ways of practicing or implementing the disclosure underthe claims.

While certain aspects of the disclosure are presented below in certainclaim forms, the inventors contemplate the various aspects of thedisclosure in any number of claim forms. For example, while only oneaspect of the disclosure is recited as a means-plus-function claim under35 U.S.C. § 112, 6, other aspects may likewise be embodied as ameans-plus-function claim, or in other forms, such as being embodied ina computer-readable medium. (Any claims intended to be treated under 35U.S.C. § 112, 6 begins with the words “means for”.) Accordingly, theapplicant reserves the right to add additional claims after filing theapplication to pursue such additional claim forms for other aspects ofthe disclosure.

What is claimed is:
 1. A mobile device comprising: a touch-sensitivedisplay; a camera; a processor; and a memory coupled to the processorand storing instructions which, when executed by the processor, causethe mobile device to: display an image, being captured via the camera,detect presence of one or more physical objects in the image, enter amode in which the mobile device receives an assistance from a user toselect a target among the one or more physical objects in the image,display a selection tool in the image to form a selected area, receive auser input for moving the selected tool or changing a size of theselected tool, in response to receiving the user input, transmitinformation associated with the selected target to an external device,and based on information of a product corresponding to the selectedtarget received from the external device, display information forconducting a transaction of the product, wherein the displayedinformation comprises information regarding the product.
 2. The mobiledevice of claim 1, wherein the memory further stores instructions which,when executed by the processor, cause the mobile device to: prompt, viathe touch-sensitive display, the user to enter the mode in which themobile device receives assistance from the user to perform thedetection.
 3. The mobile device of claim 2, wherein the memory furtherstores instructions which, when executed by the processor, cause themobile device to: provide, via the touch-sensitive display, the userwith the selection tool in a perspective to form the select area, andreceive a user input for selecting target corresponding to the selectedarea by using the selection tool.
 4. The mobile device of claim 3,wherein the memory further stores instructions which, when executed bythe processor, cause the mobile device to: receive a user input formoving the selected area, and change another target based on the userinput.
 5. The mobile device of claim 3, wherein the memory furtherstores instructions which, when executed by the processor, cause themobile device to: receive a user input for changing a size of theselected area, and detect the target based on the selected area resized.6. The mobile device of claim 1, wherein the displayed informationcomprises an advertisement regarding the product and another similarproduct relating to the product.
 7. The mobile device of claim 1,wherein the displayed information comprises at least one of purchaseinformation, price information or vendor information of the product andanother similar product relating to the product.
 8. The mobile device ofclaim 1, wherein the memory further stores instructions which, whenexecuted by the processor, cause the mobile device to: detect thepresence of the one or more targets in the selected area based ongeolocation information of the mobile device.
 9. A method for selectinga target representing a physical object, via a mobile device, the methodcomprising: displaying an image, being captured via a camera of a mobiledevice; detecting presence of one or more physical objects in the image;entering a mode in which the mobile device receives an assistance from auser to select a target among the one or more physical objects in theimage; display a selection tool in the image to form a select area;receiving a user input for moving the selected tool or changing a sizeof the selected tool; in response to selecting the target, transmittinginformation associated with the selected target to an external device;and based on information of a product corresponding to the selectedtarget received from the external device, displaying information forconducting a transaction of the product, wherein the displayedinformation comprises information regarding the product and anothersimilar product relating to the product.
 10. The method of claim 9,wherein the method further comprises: prompting, via a touch-sensitivedisplay of the mobile device, the user to enter the mode in which themobile device receives assistance from the user to select the target.11. The method of claim 10, wherein the method further comprises:providing, via the touch-sensitive display, the user with the selectiontool in a perspective to form a select area; and receiving a user inputfor selecting the target corresponding to the selected area by using theselection tool.
 12. The method of claim 11, wherein the method furthercomprises: receiving the user input for moving the selected area; andchanging another target based on the user input.
 13. The method of claim11, wherein the method further comprises: receiving the user input forchanging a size of the selected area; and detecting the target based onthe selected area resized.
 14. The method of claim 9, wherein thedisplayed information comprises an advertisement regarding the productand another similar product relating to the product.
 15. The method ofclaim 9, wherein the displayed information comprises at least one ofpurchase information, price information or vendor information of theproduct and another similar product relating to the product.
 16. Themethod of claim 9, wherein the method further comprises: detecting thepresence of the one or more targets in the selected area based ongeolocation information of the mobile device.